Lexical Category Acquisition as an Incremental Process

Psycholinguistic studies suggest that early on children acquire robust knowledge of the abstract lexical categories such as nouns, verbs and determiners (e.g., Gelman & Taylor, 1984; Kemp et al., 2005). Children’s grouping of words into categories might be based on various cues, including the phonological and morphological properties of a word, the distributional information about its surrounding context, and its semantic features. Among these, the distributional properties of the local context of a word have been shown to be a reliable cue for the formation of the lexical categories (Redington et al., 1998; Mintz, 2003). Several computational models have used distributional information for categorizing words (e.g. Brown et al., 1992; Schütze, 1993; Redington et al., 1998; Clark, 2000; Mintz, 2002). The majority of these models use iterative, unsupervised methods that partition the vocabulary into a set of optimum clusters (e.g., Brown et al., 1992; Clark, 2000). The generated clusters are intuitive, and can be used in different tasks such as word prediction and parsing. Moreover, these models confirm the learnability of abstract word categories, and hint at distributional cues as a useful source of information for this purpose. The process of learning word categories by children is necessarily incremental. Human language acquisition is bounded by memory and processing limitations, and it is implausible that humans process large volumes of text at once and induce an optimum set of categories. Efficient online computational models must be developed to investigate whether the distributional information is equally powerful in an online process of word categorization. There have only been a few previous attempts at applying an incremental method to category acquisition. The model of Cartwright & Brent (1997) uses an algorithm which incrementally merges word clusters so that a Minimum Description Length criterion for a template grammar is optimized. The model treats whole sentences as contextual units, which sacrifices a degree of incrementality, as well as making it less robust to noise in the input. The model proposed by Parisien et al. (2008) uses a Bayesian clustering algorithm that can cope with ambiguity, and shows the developmental trends observed in children (e.g. the order of acquisition of different categories). However, their fully Bayesian implementation is computationally expensive. Moreover, when measuring the similarity between two contexts, the model is sensitive to mismatches between any pair of context features, which results in the creation of sparse clusters. To overcome the problem, they introduce a bootstrapping mechanism which improves the performance, but adds substantially to the computational load. We propose an efficient incremental model for clustering words into categories based on their local context. Each word of a sentence is processed and categorized individually based on the similarity of its content (the word itself) and its context (the surrounding words) to the existing clusters. We test our model on a corpus of child-directed speech from CHILDES (MacWhinney, 2000). Over time, the model learns a finegrained set of word categories that are intuitive and can be used in a variety of tasks. We evaluate our model on a word prediction task, where a missing word is guessed based on its context. We also use our model to infer the semantic properties of a novel word based on the context it appears in. In both tasks, we show that our induced categories outperform the part of speech tags used for annotating the corpus.